CI: Free more disk space with free-disk-space script #134151

whiteio · 2024-12-10T23:43:49Z

Attempting to move the free-disk-space github action into the repo as it isn't currently maintained. This involves moving the script within free-disk-space into the repo.

I'm opening this as a draft to verify that the saved space is the same as the original action. Once that is verified, I'll look into integrating these two PRs to improve it:

Changes so far:

Took the script from the free-disk-space repo and moved it into src/ci/scripts/
Removed unnecessary if statements which were used to check for github action inputs which we don't need to check
Swapped to using this script within the rust repo, instead of using the jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be GitHub action

See this issue for more info: rust-lang/infra-team#183

…cript

rustbot · 2024-12-10T23:43:58Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Mark-Simulacrum (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

whiteio · 2024-12-11T23:02:29Z

Confirmed that the script behaves the same when moved to the repo, still trying to figure out if jlumbroso/free-disk-space#26 makes a tangible difference.

Currently, free disk space can clean up 36GB with the changes up to this point, so a 10GB improvement so far.

rust-log-analyzer · 2024-12-12T00:28:23Z

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

/usr/local/share/chromium
chrome-linux

********************************************************************************
=> Chromium: Saved 0B

/usr/local/lib/node_modules
@bazel
corepack
---
********************************************************************************

/usr/share/dotnet
LICENSE.txt
ThirdPartyNotices.txt
host
metadata
packs
sdk
---
********************************************************************************
=> .NET runtime: Saved 149KiB
********************************************************************************

src/ci/scripts/free-disk-space.sh: line 9: cd: /opt/ghc: No such file or directory
host
metadata
packs
sdk
---
********************************************************************************
=> Haskell runtime: Saved 0B
********************************************************************************

dpkg: no packages found matching gcloud-*
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  aspnetcore-runtime-6.0 aspnetcore-runtime-7.0 aspnetcore-runtime-8.0
  aspnetcore-runtime-6.0 aspnetcore-runtime-7.0 aspnetcore-runtime-8.0
  aspnetcore-targeting-pack-6.0 aspnetcore-targeting-pack-7.0
  aspnetcore-targeting-pack-8.0 azure-cli clang-12 clang-tidy-12
  clang-tools-12 debsuryorg-archive-keyring dictionaries-common
  dotnet-host dotnet-hostfxr-6.0 dotnet-hostfxr-7.0 dotnet-hostfxr-8.0
  dotnet-runtime-6.0 dotnet-runtime-7.0 dotnet-runtime-8.0
  dotnet-runtime-deps-6.0 dotnet-runtime-deps-7.0 dotnet-runtime-deps-8.0
  dotnet-sdk-6.0 dotnet-sdk-7.0 dotnet-sdk-8.0 dotnet-targeting-pack-6.0
---
Removing php8.3-sybase (8.3.14-2+ubuntu20.04.1+deb.sury.org+1) ...
Removing php8.2-sybase (8.2.26-3+ubuntu20.04.1+deb.sury.org+1) ...
Removing google-chrome-stable (131.0.6778.85-1) ...
Removing google-cloud-cli-anthoscli (502.0.0-0) ...
/var/lib/dpkg/info/google-cloud-cli-anthoscli.prerm
Cleaning Google Cloud CLI files...
44238
Cleaning Google Cloud CLI manuals...
Removing google-cloud-cli (502.0.0-0) ...
Removing google-cloud-cli (502.0.0-0) ...
/var/lib/dpkg/info/google-cloud-cli.prerm
Removing php8.2-pspell (8.2.26-3+ubuntu20.04.1+deb.sury.org+1) ...
Removing libcgi-fast-perl (1:2.15-1) ...
Removing libhtml-template-perl (2.97-1) ...
Removing libcgi-pm-perl (4.46-1) ...
---

COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 2.784 Building wheels for collected packages: reuse
#13 2.785   Building wheel for reuse (pyproject.toml): started
#13 3.050   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.051   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132720 sha256=026f3bb0f1aa8090b861fd0a0939cb1a782396d84c8aab7875096557d637a0f6
#13 3.051   Stored in directory: /tmp/pip-ephem-wheel-cache-89ugijb8/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.054 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.450 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.451 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 3.977 Collecting virtualenv
#13 3.977 Collecting virtualenv
#13 4.017   Downloading virtualenv-20.28.0-py3-none-any.whl (4.3 MB)
#13 4.089      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 61.4 MB/s eta 0:00:00
#13 4.144 Collecting platformdirs<5,>=3.9.1
#13 4.148   Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
#13 4.184 Collecting filelock<4,>=3.12.2
#13 4.207 Collecting distlib<1,>=0.3.7
#13 4.212   Downloading distlib-0.3.9-py2.py3-none-any.whl (468 kB)
#13 4.219      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 86.1 MB/s eta 0:00:00
#13 4.219      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 86.1 MB/s eta 0:00:00
#13 4.299 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 4.484 Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.28.0
#13 DONE 4.6s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      280512 kB
DirectMap2M:     7059456 kB
DirectMap1G:    11534336 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
downloading https://static.rust-lang.org/dist/2024-11-27/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
---
fmt check
fmt: checked 5706 files
tidy check
tidy: Skipping binary file check, read-only filesystem
##[error]tidy error: /checkout/src/ci/scripts/free-disk-space.sh:164: line longer than 100 chars
##[error]tidy error: /checkout/src/ci/scripts/free-disk-space.sh:176: line longer than 100 chars
removing old virtual environment
creating virtual environment at '/checkout/obj/build/venv' using 'python3.10'
Requirement already satisfied: pip in ./build/venv/lib/python3.10/site-packages (24.3.1)
All checks passed!
checking C++ file formatting
some tidy checks failed
some tidy checks failed
Command has failed. Rerun with -v to see more details.
  local time: Thu Dec 12 00:28:11 UTC 2024
  network time: Thu, 12 Dec 2024 00:28:11 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

larziwau · 2024-12-13T12:40:09Z

e447a4f (#134151)
does this commit really make a difference?

whiteio added 2 commits December 10, 2024 23:26

Add free-disk-space script

957a266

Replace usage of free-disk-space github action with free-disk-space s…

a0665ac

…cript

rustbot assigned Mark-Simulacrum Dec 10, 2024

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Dec 10, 2024